Parallel methods for the generation of partitioned inverted files

The generation of inverted indexes is one of the most computationally intensive activities for information retrieval (IR) systems: indexing large multigigabyte text databases can take many hours or even days to complete. We examine the generation of partitioned inverted files in order to speed up the process of indexing. We describe the components of PLIERS, the system used to index the documents and how these components can be re-configured to generate indexes with different types of partitioning. Two types of index partitions are investigated: TermId and DocId. Two types of build are investigated: local and distributed. The results from runs on both partitioning and build methods are compared and contrasted, concluding that DocId is the more efficient method.